Usenet 1993 July

home *** CD-ROM | disk | FTP | other *** search

/ Usenet 1993 July / InfoMagic USENET CD-ROM July 1993.ISO / sources / unix / volume11 / test.el / part03 < prev next >

Wrap

Internet Message Format | 1987-09-08 | 63.7 KB

Path: uunet!rs From: rs@uunet.UU.NET (Rich Salz) Newsgroups: comp.sources.unix Subject: v11i038: Test system for GNU Emacs, Part03/03 Message-ID: <1503@uunet.UU.NET> Date: 10 Sep 87 03:54:52 GMT Organization: UUNET Communications Services, Arlington, VA Lines: 1722 Approved: rs@uunet.UU.NET Submitted-by: "Mark A. Ardis" <maa@sei.cmu.edu> Posting-number: Volume 11, Issue 38 Archive-name: test.el/Part03 #! /bin/sh # This is a shell archive, meaning: # 1. Remove everything above the #! /bin/sh line. # 2. Save the resulting text in a file. # 3. Execute the file with /bin/sh (not csh) to create: # MANIFEST # test.texinfo export PATH; PATH=/bin:/usr/bin:$PATH echo shar: "extracting 'MANIFEST'" '(1149 characters)' if test -f 'MANIFEST' then echo shar: "will not over-write existing file 'MANIFEST'" else sed 's/^X//' << \SHAR_EOF > 'MANIFEST' XFile Archive Description X---- ------- ----------- XMANIFEST 1 This file Xbox.script 2 Example test script Xexample.texinfo 2 Appendix part of user manual Xhooks.el 2 Example hooks for customization Xtest.el 2 Root of test package Xtest.texinfo 1 Main part of user manual Xtst-achieve.el 2 Achieve-state routines Xtst-analyze.el 2 Analyze test results Xtst-annotate.el 2 Annotate package with results Xtst-capture.el 2 Capture-state routines Xtst-display.el 3 Display test results Xtst-equal.el 3 Test states for equality Xtst-inequal.el 3 Modifications to notion of equality Xtst-instrument.el 3 Instrument package for testing Xtst-utilities.el 2 Miscellaneous utilities Xuser.texinfo.el 2 Wrapper for hard-copy user manual SHAR_EOF if test 1149 -ne "`wc -c < 'MANIFEST'`" then echo shar: "error transmitting 'MANIFEST'" '(should have been 1149 characters)' fi fi echo shar: "extracting 'test.texinfo'" '(60887 characters)' if test -f 'test.texinfo' then echo shar: "will not over-write existing file 'test.texinfo'" else sed 's/^X//' << \SHAR_EOF > 'test.texinfo' X@setfilename test X@node top, introduction, , (dir) X@ifinfo XThere should be more nodes than listed below, but at least the chapters are Xisolated. X@menu X* introduction:: Purpose of the package. X* starting:: How to get started. X* testing:: Testing a package. X* instrumenting:: Instrumenting to measure testing effectiveness. X* analyzing:: Analyzing the results of testing. X* displaying:: Displaying the results of analysis. X* running:: Running a testscript in batch. X* epilogue:: Final words. X* example:: An example package to test. X@end menu X@end ifinfo X@node introduction, starting, , top X X@chapter Introduction X XThis document explains how to use the Test package to test GNU Emacs Xpackages. It is an extension of Chapter 22 [Compiling and Testing XPrograms] of the GNU Emacs Reference Manual. X X@section Purpose of the Package X XThis package provides tools to aid in developing regression tests for Emacs Xpackages and to assess the efficacy of those tests. GNU Emacs packages, Xwritten in a dialect of Lisp, provide extensions to the basic text editing Xfeatures of Emacs. Most of these packages perform state-changing Xoperations. For example, a package may create new buffers, modify text, Xchange the locations of the cursor and various marks, write files or modify Xanything else accessible to Emacs (which is just about everything in Unix). XTraditional Lisp debugging strategies are not sufficient to adequately test Xa package that performs extensive state-manipulation. The Test package Xprovides a strategy that is. X X@section Test Approach X XThe Test system is designed to support structural (or ``white-box'') Xtesting. (Another commonly used test approach is functional, or X``black-box'' testing.) It automates the process of running a number of Xtests cases designed to analyze ``coverage'' of the package (that is, find Xparts of the code that are not exercised) and ``expression'' (that is, what Xvalues are returned by each expression, and which expressions always Xreturned the same values). X X@section Testing Method X XTest provides mechanisms to ease the creation and execution of sets of Xtests. Some of the features are designed to simplify testing activity, Xwhile others are designed to aid the analysis of testing. X XTest supports coverage testing. That is, it provides instrumentation Xmechanisms designed to check for inadequacies of testing of the following Xtwo forms: X@itemize @bullet X@item XFailure to execute an expression. X@item XFailure to produce more than one value for an expression. X@end itemize XIn each case, the fault may be in the tests (i.e., not enough testing was Xperformed), or the fault may be in the program (i.e., an ``unnecessarily Xcomplex'' program has been written). (There is also the possibility that Xno fault exists, such as the use of ``constant'' functions for readability Xor style. In practice these cases are easy to identify and ignore.) X XYou should use the following strategy in creating tests with this Xsystem: X@enumerate X@item XCreate a simple set of tests that appear to be necessary to test the basic Xfunctionality of the package. X@item XAdd additional tests to ensure that every expression is executed, and that Xeach expression produces at least two values over the entire set of tests. X@item XRun the set of tests with instrumentation to check for inadequacies. X@item XIf there are reported inadequacies, then correct the code or improve the Xtesting, and go back to the previous step. X@end enumerate XThere is no guarantee that a set of tests that passes these coverage checks Xwill find all errors. But, such a set of tests has the property that every Xexpression in the code has been exercised in a minimally significant way X(i.e., every expression has produced at least two different values). In Xpractice, a great number of errors can be detected with this method. X XThe importance of coverage testing increases as a package is modified. XChanges to one part of a package may have unexpected effects on other X(unconsidered) parts. Coverage testing ensures that all parts of a package Xare tested, at least minimally. Coverage tests also serve as a Xspecification of the functionality of a package. X XFigure 1 illustrates the overall flow of events using this test Xapproach. X X@group X@example X +------------+ +--------------+ X | Instrument | | Generate | X | o Control | | o Test Cases | X | o Data | | o Script(s) | X +------------+ +--------------+ X \ / ^ X \ / | X \ / | X \ / | X \ / | X \ / | X +-------------+ | X | Execute | | X | the | | X | Test(s) | | X +-------------+ | X | | X | | X +-------------+ | X | Analyze, | | X | Display | | X | Results | | X +-------------+ | X | | X | | X ^ | X / \ | X / \ No | X < Done? >-------------+ X \ / X \ / X \ / X v X | Yes X |(Test Goals Achieved) X V X X X X@end example X@center Figure 1. General Test Algorithm X@end group X X@section Overview X XThe Test product consists of four major subsystems: X@itemize @bullet X@item XRegression Subsystem -- a set of functions to support regression testing of XEmacs packages. X@item XCoverage Subsystem -- a set of functions to support coverage testing of XEmacs packages. X@item XAnnotation Subsystem -- a set of functions to provide support for creating, Xusing and displaying information about Emacs Lisp code. X@item XEquality Subsystem -- a set of functions used by the testing Xsubsystems that provide different interpretations of equality. X@end itemize X XThe following sections explain the use of these functions to support Xthe steps in the testing process: X@itemize @bullet X@item XGetting Started X@item XInstrumenting a package X@item XTesting a Package X@item XAnalyzing Test Results X@item XDisplaying Test Results X@end itemize X XThe system is designed to test Emacs-Lisp programs, and is implemented as a Xminor mode of Emacs-Lisp mode (See Section 28.1 [Minor Modes] of the GNU XEmacs Reference Manual). X X@node starting, testing, introduction, top X@chapter Getting Started X XThis section describes how to begin using the Test package to test XEmacs packages. X X@section Loading and Initialization X XTo start testing you probably want a fresh Emacs session with no extraneous Xbuffers or processes hanging around. Non-essential extras will just slow Xdown the state capture and achieve functions. If you have a busy X@file{.emacs} file you may want to keep a trimmed back copy just for test Xsessions. X XOnce in Emacs visit the file containing Emacs-lisp code to be tested, Xthen enter: X@example XM-x load-file Xtest.el XM-x test-mode X@end example XThe mode line at the bottom of your window should show ``(Emacs-lisp XTest)'' in the mode field. You are now in Test mode. X XProbably the first thing to do at this point is to use @samp{C-c n} to Xgenerate a new buffer containing a test script template. This is a good Xstarting point for creating a new test script; the template is shown in Xsection 2.3 of this document. The task now confronting you is to develop a Xtest suite that will exercise your package throughly, not only to test it Xnow but also to have an automated regression test. The remainder of this Xdocument will guide you through this task. X XFigure 2 illustrates the overall flow of data in the Test system. X X@group X@example X +-------------+ X |Source Text | X +-------------+ X \ X \ X \ X Instrument X / X / X / X +-------------+ +-------------+ X |Instrumented | | Test Cases | X | Text | +-------------+ X +-------------+ / X \ / X \ / X \ / X \ / X Execute X \ X \ X \ X \ X +-------------+ X |Test Results | X +-------------+ X / ^ X / | X / | X / v X Display Analyze X X X X@end example X@center Figure 2. Test Data Flow X@end group X X@section Test Mode X XThe package `@samp{test.el}' provides several functions to the user: X@table @samp X@item C-c n XStart a new test script X@code{ (tst-new-script-buffer)} X@item C-c i X(Re-)instrument a test program buffer @code{ (tst-instrument)}. X@item C-c x XTest the program, using the test script @code{ (tst-execute)}. X@item C-c c XRecord a GNU Emacs state for later use X@code{ (tst-capture-state)}. X@item C-c a XAchieve a previously saved GNU Emacs state X@code{ (tst-achieve-state)}. X@item C-c C-a XAchieve a state previously saved-to-file X@code{ (tst-achieve-state-from-file)}. X@item C-c C-c XCapture current emacs state to file X@code{ (tst-capture-state-to-file)}. X@item C-c C-r XRead a previously saved-to-file state into a variable X@code{ (tst-read-state-from-file)}. X@item C-c C-w XWrite a state previously captured in a variable to a file X@code{ (tst-write-state-to-file)}. X@item C-c f XAnalyze (filter) the results of running the test(s) @code{ (tst-analyze)}. X@item C-c d XDisplay the annotations about the test @code{ (tst-display-mode)}. X@item C-c q XExit test mode @code{ (test-mode)}. X@end table X X@section Contents of the *test script* Buffer X X X@example X(defun test-script () X (interactive) X ; Local Variables X (let (pre post actual) X ; Body X (switch-to-buffer foo) X (erase-buffer) X (insert-file foo) X (goto-char (point-min)) X (emacs-lisp-mode) X (test-mode) X (tst-instrument) X ; 1st test run X (tst-read-state-from-file 'post expected-state-file1) X (tst-achieve-state-from-file 'pre initial-state-file1) X (foo args) X (tst-capture-state 'actual nil nil) X;;; If you have a lot of tests, consider a while loop X ) ; let X) ; defun X@end example X X@section The Example -- box.el X XThe package `@samp{box.el}' provides two functions to the user: X@code{box-region} and @code{unbox}. XThe first function encloses the current region with a box: X X@group X@example X/********************************************\ X* A boxed region * X* may have lines of text of varying lengths. * X* A side effect of boxing is to normalize * X* the length of lines. * X\********************************************/ X@end example X@end group X XThe second function undoes the effect of the first by removing any boxes Xwithin the region. This simple package will be used in this document as an Xexample of a package to be tested using the Test package. X X@node testing, instrumenting, starting, top X@chapter Testing Your Package X XAs mentioned at the beginning of this document, Emacs packages are Xgenerally state changing operations. You invoke the package with a session Xthat is in some arbitrary @i{initial state}. The package modifies Xselected aspects of this @i{initial state} in a defined manner and leaves Xthe session in some @i{final state}. The @dfn{state} of a GNU Emacs Xsession consists of information on buffers, windows, processes, and other Xglobal information. The components of the state that are used in the Test Xpackage are listed later in this section. X X@dfn{Regression Testing} is a technique for verifying that changes in a Xpackage do not unintentionally degrade the quality of the code. Any Xchanges in one part of the package should not alter the functionality of Xother parts of the package. State information plays two roles in Xregression testing of Emacs packages. These are: X@enumerate X@item Xinitiating the run of a package in a consistent and Xpre-determined @dfn{initial state}, and X@item Xcomparing the @dfn{actual final state} after the completion of a Xpackage to a @dfn{desired final state}. X@end enumerate XThe regression testing components of @samp{test.el} allow you to X@dfn{capture} session states for later use, @dfn{achieve} these states to Xset up a test run, and test the @dfn{equality} of states for package Xverification. X XThe following example demonstrates these steps for regressions testing an Xarbitrary package. Assume that a working version of some package exists. XNow suppose you wish to add some new function that requires some Xmodifications in the original code. After adding the new function you need Xto verify that the original functionality of the package has not changed. XIn other words, you need to verify that the code has not @dfn{regressed} Xfrom its original functionality. To perform this testing, you X@dfn{capture} the initial state of a session, run the original package, and X@dfn{capture} the final state. Following this, you @dfn{achieve} the Xinitial state that you previously captured, run the new version of the Xpackage, and perform an @dfn{equality} test to compare the new final state Xto the one that was captured after the run of the original package. The Xstates should be equivalent if the test is successful. X XTwo extensions to this state comparison paradigm are: X@enumerate X@item XTesting for partial equivalence - Each Emacs package is supposed to change Xcertain aspects of a session state, but leave others unchanged. For Xexample, a package may change the text of the current buffer but should Xleave the window configuration of the session intact. Partial equivalence Xtesting consists of the following steps: X@enumerate X@item XCapture the initial state. X@item XRun the function to be tested. X@item XCompare the final state to the initial state. The states should be Xequivalent in all areas except those that the test function is supposed to Xchange. X@end enumerate X@item XTesting functions that undo the effects of other functions - Some emacs Xpackages have a function that perform some state change and a corresponding Xfunction that undoes that change. Testing these types of packages consists Xof the following steps: X@enumerate X@item XCapture the initial state. X@item XRun the function that performs the state change. X@item XRun the function that undoes the state change. X@item XCompare the final state to the initial state. They should be equivalent. X@end enumerate X@end enumerate XThe remainder of this section describes the elements of the Test Xpackage that implement these aspects of regression testing. XIn summary, these are: X@itemize @bullet X@item X@code{capture} - saves the current state of the Emacs session Xin either a bound variable or UNIX file. X@item X@code{partial capture} - save part of the current state of the XEmacs session. Some or all global variables may be excluded, and a subset Xof buffers may be selected. X@item X@code{achieve} - sets the state of the current session to one Xthat is either in a bound variable or a UNIX file. X@item X@code{equality} - compares two emacs sessions states for Xequivalence. X@item X@code{partial equality} - compares selected aspects of two emacs Xsession states for equivalence. X@item X@code{inequality} - using user defined hooks, compares two Xsession states within certain degrees of inequality (e.g. compares buffer Xcontents ignoring whitespace). X@end itemize X X@section Capturing and Achieving Emacs States X X@subsection GNU Emacs State Definition X XThe @dfn{capture} and @dfn{achieve} functions of @samp{test.el} represent Xan Emacs state as the hierarchy of data shown below. X@itemize @bullet X@item X@code{Global Bound Symbols} - Most of the information about an XEmacs session is contained in the values of symbols global to the session. XIn conventional programming language terms, this session information can be Xviewed as the values of global variables. Some examples of common global Xbound symbols are: X@itemize @bullet X@item X@samp{global-abbrev-table} - the list of abbreviations global to Xthe session and their meanings. X@item X@samp{global-map} - a vector describing the mappings of all keys Xto Emacs commands. X@item X@samp{debug-on-error} - a flag that indicates that the Emacs Xdebugger should be called on an error (if non-nil). X@end itemize X@item X@code{Buffer Information} - Each Emacs session contains one or Xmore buffers. The capture functions store the following information for Xeach active buffer: X@itemize @bullet X@item Xbuffer name - a unique name that identifies the buffer. X@item Xfile name - if the buffer is associated with a UNIX file, the Xname of that file. X@item Xpoint - the value of the cursor point in the buffer. X@item Xmark - the value of the buffer's mark. X@item Xcontents - the contents of the buffer represented as a text string. X@item Xmodified - a flag indicating if the buffer has been modified Xsince last saved. X@item Xlocal map - the local key map for this buffer. X@item Xlocal bound symbols - symbols (variables) that are local to this Xbuffer (e.g., the @samp{current-mode} of the buffer). X@end itemize X@item X@code{Window Information} - At any time an Emacs session has Xone or more windows visible. The capture functions store the following Xinformation for each window: X@itemize @bullet X@item Xedges - the upper left and lower right corner of the window in Xscreen coordinates. X@item Xbuffer - the buffer occupying this window. X@item Xstart - the integer position in the buffer where display starts Xin the respective window. X@item Xpoint - the integer position in the buffer where the point for Xthis window sits. X@item Xcurrent flag - a flag that is non-nil if this is the window in Xwhich the screen cursor currently is located. X@end itemize X@item X@code{Process Information} - An Emacs session may have invoked Xseveral UNIX processes during its lifetime. These processes may have Xcompleted or may still be running. Two common processes that run during an XEmacs sessions are the @samp{shell} and the process that displays the time, Xdate and system load on the @dfn{mode line}. The @samp{Test} capture Xfunctions store the follow information for each process: X@itemize @bullet X@item Xbuffer - the name of the buffer to which the process is attached X(nil if the process is not related to a buffer, e.g., the display time Xprocess). X@item Xprocess mark - the marker for the end of the last output from Xthe process. X@item Xcommand - the UNIX command that invoked the process. X@item Xexit status - if the process has completed, the UNIX return code Xfrom that process. X@item Xfilter - the name of an Emacs lisp function that receives all Xoutput of the process. X@item Xname - a unique name of the program invoked in the process. X@item Xsentinel - the name of an Emacs lisp function that is called Xwhen the process changes state (e.g. completes). X@item Xstatus - indicates whether the process is running, complete, Xetc. X@end itemize X@end itemize X XA significant item that is excluded from this state information is the Xcurrent definition of bound functions. Recall that an Emacs user can Xdefine any new function with a @samp{defun} and load that function. XFurthermore, the user can redefine any existing function (e.g. X@code{forward-char} !) in this manner. Saving the current definition of Xall bound functions would result in a huge state vector, and is therefore Xnot done. X X@subsection Capturing an Emacs Session State XWhen regression testing an Emacs package, you will mainly use the state Xcapture functions of Test in two ways: X@enumerate X@item Xto save a known @dfn{initial state} to use as the base state for Xa run of an Emacs package, and X@item Xto save a @dfn{final state} to later compare to the state of the Xsession after the run of a modified package. X@end enumerate X X@subsubsection Commands for Capturing State X X@table @samp X@item tst-capture-state-to-file XWrites the current state of the Emacs session to a file. You are prompted Xfor the name of the file, the @dfn{exclude-variables} and the X@dfn{buffer-list}. Refer to the @code{Usage Notes} below for details on Xthese last two items. If the file you specify already exists it is Xoverwritten. X@item tst-write-state-to-file XWrites a state that is saved in a bound variable to a file. You are Xprompted for the state variable name (which was created with X@samp{tst-capture-state}) and the name of the file. If the file already Xexists it is overwritten. X@item tst-capture-state XSaves the state of the Emacs session in a state variable. You are prompted Xfor the name of the file, the @dfn{exclude-variables} and the X@dfn{buffer-list}. Refer to the @code{Usage Notes} below for more details Xon these last two items. If the bound variable you specify already exists, Xit is overwritten. X@end table X X@subsubsection Usage Notes X XThe amount of information saved by @samp{tst-capture-state-to-file} and X@samp{tst-capture-state} is controlled by your responses to the command Xprompts. These are: X@table @samp X@item List of buffers to capture: XThere are three possible responses to this prompt: X@enumerate X@item X@samp{nil} to indicate that state information on all buffers should be Xsaved (the default). X@item XA list of of buffer names indicating for which buffers state information Xshould be saved. For example, to save only the states of buffers X@samp{bar} and @samp{foo} you enter: X@example X ("bar" "foo") X@end example X@item XA singleton list that is the name of a non-existent buffer if you do not Xwish to save any buffer state information. For example, if no buffer X@samp{xxx} exists you could enter X@example X ("xxx") X@end example Xto specify that capture not save any buffer state information. X@end enumerate X@item List global vars to exclude: XThere are three possible responses to this prompt: X@enumerate X@item X@samp{nil} to capture all global variables. X@item X@samp{all} to exclude all global variables from the captured Xstate. X@item XA list of global variable names to exclude from the captured state. For Xexample, if you enter X@example X ("obarray" "values") X@end example Xthe two global variables, @samp{obarray} and @samp{values}, are not Xcaptured. This value is the default for the prompt since these two Xvariables have very large values that are rarely of use in testing. X@end enumerate X@end table X X@subsection Achieving an Emacs State X XWhen using Test you will mainly use the achieve state functions to return Xyour session to a known @dfn{initial state}. With these functions you can Xmake repeated test runs by simply achieving the @dfn{initial state} between Xeach run. X X@subsubsection Commands for State Achieve X@table @samp X@item tst-read-state-from-file XReads a session state from a file into a bound symbol. You are prompted Xfor the name of the file and the name of the bound symbol. The file should Xhave been created by @samp{capture-state-to-file} or X@samp{write-state-to-file}. If the bound symbol already exists, it will be Xoverwritten. X@item tst-achieve-state-from-file XSets the state of the emacs session to that saved in a file. You are Xprompted for the name of the file. The file should have been created by X@samp{capture-state-to-file} or @samp{write-state-to-file}. X@item tst-achieve-state XSets the state of the emacs session to that saved in a bound symbol. You Xare prompted for the name of the bound symbol. The symbol must have been Xcreated by @samp{tst-capture-state}. X@end table X X@subsubsection Usage Notes X XAchieving the state of a previous Emacs session is a non-trivial task that Xcan have many unexpected side-effects. These are listed below: X@enumerate X@item XIf you want @samp{achieve} to preserve all existing buffers that were not Xpart of the state you are achieving, you may set the variable X@samp{tst-achieve-buffers-nondestructively} non nil. X@item XAchieving a previous state completely resets your Emacs session. Any Xinformation in your current state will be overwritten by the achieved Xstate. For example, any changes in buffers will not exist after the Xachieve process. You should, therefore, make sure that you save any Xchanges to disk before running achieve. X@item XIf you are not careful you may find that previous versions of disk files Xoverwrite current changed versions. The following scenario demonstrates Xthis problem. X@enumerate X@item XSuppose you have a file @samp{mytest} that is in a buffer of an emacs Xsession at the time you run @code{tst-capture-state-to-file}. X@item XNow suppose you make changes to this file over a period of a few days, and Xthen run @code{tst-achieve-state-from-file}. X@item XYour emacs session will now contain the old version of the file. X@item XIf written to disk, this old version will overwrite the current version of X@samp{mytest}. X@end enumerate X XYou can avoid this problem by testing packages only on dummy test files, or Xusing a special test directory with copies of files that are permanently Xstored in other directories, or by not capturing buffer @samp{mytest} and Xsetting @code{tst-achieve-buffers-nondestructively} to @code{t}. X X@item XThe state of an emacs session is partially dependent on the state of your Xentire UNIX session. Therefore, the achieve state functions may not be Xable to restore your state to the exact one that was captured. For Xexample, a process can not be started if the run image for the program no Xlonger exists on disk. When testing and comparing states, it is best to Xrely only on the more common aspects of an Emacs session (e.g. buffers, Xwindows). X@item XAs explained earlier, the capture functions do not save all aspects of the Xsession. A significant example is the current definition of all bound Xfunctions. Therefore, the achieve functions will not restore these aspects Xof the session state. X@end enumerate X X@section Testing for equality X XThe equality package of @b{Test} compares two Emacs states for equivalence Xby invoking special equality functions on each of the data components Xcaptured in the state. Each of these equality functions compares only a Xsmall part of the entire state. The results of these comparisons are Xwritten into the buffer @samp{*equal-log*}. If two states are not equal Xthen you can determine where components differ by looking at this buffer. X XThe section @b{Equality Functions} provides a list of all of the Xequivalence testing functions. The section @b{Partial Equality} explains Xhow you can accomplish testing a subset of a state. This is useful if you Xare only interested in differences between certain aspects of state, such Xas only the contents of buffers, but you do not care to hear about Xdifferences in other components, such as windows. The section X@b{Inequality} explains how to add hooks to the equality functions. These Xhooks are called by the equality functions when two components are not Xequal but before a result is returned. A hook that you write can be Xinserted at this point and change a ``not equal'' result into an ``equal.'' X X@subsection Equality Functions X XThe functions to compare states for equality are divided into two Xcategories. One set of functions compares complete states, the second set Xof functions only uses the buffer components. X XThe following state equality functions are defined for interactive use. X@table @code X@item tst-equ-state XCompares two states for equality. Each component in the first state is Xcompared to its corresponding component in the second state. X@item tst-equ-sessions XCompares the @b{sessions} component of two states. The sessions components Xcontains all global variables. These includes those that define key-maps Xand syntax tables. X@item tst-equ-buffers XCompares the @b{buffers} components of two states. Compares all of the Xbuffers in one state with those of a second state. X@item tst-equ-windows XCompares the @b{windows} components of two states. X@item tst-equ-process XCompares the @b{process} components of two states. X@end table X XThe following functions operate on just the @b{buffer} components of Xstates. Objects of this type can be extracted from state objects by the Xfunction @code{tst-equ-find-buff-with-name}. X@table @code X@item tst-equ-buffer-state XCompares two buffers for equality. The following components are checked for Xequality: @b{point}, @b{mark}, @b{contents}, @b{modified}, @b{file}, X@b{local-variables}. X@item tst-equ-point XCompares the @b{point} component of two buffers. X@item tst-equ-mark XCompares the @b{mark} component of two buffers. X@item tst-equ-file XCompares the @b{file} component of two buffers. X@item tst-equ-modified XCompares the @b{modified} component of two buffers. X@item tst-equ-contents XCompares the @b{contents} component of two buffers. Contents are compared Xas two single strings. X@item tst-equ-contents-line XCompares the @b{contents} component of two buffers. Contents are viewed as Xcomposed of lines of text, and compared line-by-line. X@item tst-equ-contents-region XCompares the @b{contents} component of two buffers. Only the contents Xwithin a region are compared. Region is delimited by point and mark. X@end table X X@subsection Partial Equality X XIf you are interested in only checking the equality of a subset of an Emacs Xstate, then there are four methods you can use. X X@table @code X@item tst-capture-state XYou can use this function to capture only a subset of state. Once this has Xbeen done, equality testing will only be performed on those parts of the Xstate that where captured. X@item tst-equ-state-functions XThis variable contains a list of equality functions to be invoked when Xcomparing two states. You can change it to test only those components of Xstate that interest you. As an example, you might want to remove X@samp{tst-equ-windows} if you are not interested in differences in windows Xbetween states. X@item tst-equ-buff-state-functions XThis variable contains a list of equality functions to be invoked when Xcomparing two @dfn{buffer states}. You can change it to test only those Xcomponents of @dfn{buffer state} that interest you. As an example, you Xmight want to remove @samp{tst-equ-point} and @samp{tst-equ-mark} if you Xare not interested in those parts of the states. X@item tst-equ-find-buffer-with-name XUse this function to obtain only the part of state that is associated with Xa particular buffer. You can then pass the result of this function to Xbuffer testing equality functions. X@end table X X@subsection Inequality X XSometimes the comparison of two objects yields ``not equal'' when wish to Xignore certain inequalities. Each of the equality functions will execute a Xhook, if one is defined, and the comparison of the two objects yields ``not Xequal.'' Within the hook you can write your own test for equality and Xchange the result of the comparison if you so desire. X XFor each equality function, there exists a hook symbol that, when defined, Xwill be run if the two objects are not equal. The name of the hook symbol Xcan be found by adding @samp{-hook} to the equality function name. The Xvalue of this hook should then be set to the name of the hook function you Xwish to execute. From within the the hook the two objects being compared Xcan be accessed by adding @samp{1} and @samp{2} to the equality function Xname. The variable @code{tst-equ-result} should be set to @samp{t} or X@samp{nil} from within your hook. X XThe following example shows a hook, called @code{ignore-zero-points}, that Xis executed whenever the result of comparing the point components of two Xbuffers is not equal. From within this function the two point components Xare accessed as @samp{tst-equ-point1} and @samp{tst-equ-point2}. The hook Xchanges the result of a point comparison from @samp{nil} to @samp{t} if one Xof the points is at position zero. X X@example X X(setq tst-equ-point-hook 'ignore-zero-points) X X(defun ignore-zero-points () X "Equality point hook, changes result to t if one point X is at position zero." X X (if (or (equal tst-equ-point1 0) (equal tst-equ-point2 0)) X (setq tst-equ-result t) ; return t if one is zero X ;else X (setq tst-equ-result nil); otherwise return nil X ) X) X X@end example X XOf particular usefulness are the hooks that are associated with the Xcontents of two buffers. The equality function @samp{tst-equ-bs-contents} Xcompares the contents of two buffers by comparing strings that contain the Xentire contents of each buffer. Should this comparison fail it is sometimes Xuseful to compare the two strings after eliminating all white-space from Xeach string. The following hook, provided with this package, does such a Xcomparison. X X@example X X(setq tst-equ-contents-hook 'ignore-white-space) X Xdefun ignore-white-space() X " Compares the contents of two buffers after removing all X white-space from each." X X (setq tst-equ-result (string-equal-less-regexp "\\s " X tst-equ-contents1 tst-equ-contents2)) X) X@end example X XA second contents hook of interest is the one associated with the X@samp{tst-equ-contents-line function}. This function compares the contents Xof two functions on a line-by-line basis. The hook for this function Xdiffers from all other hooks, in that it is called once per line, instead Xof once per call to its associated function. The two lines are accessed as X@samp{tst-equ-line1} and @samp{tst-equ-line2}. X X X@section Testing example X XThe example that follows illustrates how to write an emacs-lisp function Xthat uses the Test package to test another function you have written. In Xthe example the two user supplied functions under test are: X@samp{box-region} and @samp{unbox}. X X@example X;;; test-box uses the Test package to test the functions box and unbox X;;; X test-box () X (interactive) X ; Local Variables X (let (initial-state ; to hold states X boxed-state X final-state X capture-buffer-list X ) X X ; First create a buffer to use the functions on X (get-buffer-create "box.junk") X (set-buffer "box.junk") X (erase-buffer) X (insert-file "/project/gnutest/test/box.junk") X X ; Save the initial state X ; Since we're only concerned with one buffer include X ; include that in the capture list X (setq capture-buffer-list '("box.junk")) X (tst-capture-state 'initial-state capture-buffer-list nil) X X ; execute the box function X (goto-char (point-min)) X (set-mark (point-max)) X (box-region nil) X X ; capture the boxed state X (tst-capture-state 'boxed-state capture-buffer-list nil) X X ; unbox and capture again X (unbox nil) X (tst-capture-state 'final-state capture-buffer-list nil) X X ; compare the initial state to the final state X ; they should be exactly equal X (if (tst-equ-state initial-state final-state X "Compare before box to after unbox") X (message "Box/unbox test passed") X (message "Box/unbox test failed")) X X ) ; let X ) ; defun X@end example X X@subsection Contents of *equal-log* X XEach of the equality functions writes the results of its comparison into a Xbuffer named ``*equal-log*''. If this buffer does not exist, then it is Xcreated. The contents of the buffer are never erased nor is the buffer Xever deleted, these actions are left to the user. X XThe buffer is created in Outline Mode. This allows selective hiding of Xcomparisons that are not of interest to the user. X XIf the result of a comparison is nil (i.e. not equal) then that entry in Xthe log is flagged with a question mark (?). In most cases, the two Xdiffering objects are also logged as in the following example. X X@example X* ?State comparison: Compare before box to after unbox X X** ?Sessions state X*** ? Global symbols X data-bytes-free 5350948 5176868 X this-command nil kill-region X data-bytes-used 383452 557532 X ?post not found in second state X post nil nil X ?file not found in second state X file X statevar state post X X X** ?Buffers state X X** Comparison of buffers named: *scratch* X*** point: "1" "1" X*** mark: nil nil X*** contents: contents equal X*** modified: nil nil X*** file: nil nil X*** local-vars: local variables are equal X X** Comparison of buffers named: *Minibuf-0* X*** point: "1" "1" X*** mark: nil nil X*** contents: contents equal X*** modified: nil nil X*** file: nil nil X*** local-vars: local variables are equal X X** Comparison of buffers named: box.el X*** point: "1" "1" X*** mark: 4480 4480 X*** contents: contents equal X*** modified: t t X*** file: nil nil X*** local-vars: local variables are equal X X** Comparison of buffers named: box.el-instrumented X*** point: "5608" "5608" X*** mark: 5608 5608 X*** contents: contents equal X*** modified: t t X*** file: nil nil X*** local-vars: local variables are equal X X** ?Comparison of buffers named: box.junk X*** ?point: "1" "234" X*** ?mark: 235 1 X*** ?contents: contents not equal X*** modified: t t X*** file: nil nil X*** local-vars: local variables are equal X X X** Processes state X X X** Window state X X*** window X**** window-edges: (0 0 10 9) (0 0 10 9) X**** window-buffer: "box.junk" "box.junk" X**** window-start: 1 1 X**** window-point: 1 1 X**** current-window: t t X X X@end example X X@node instrumenting, analyzing, testing, top X@chapter Instrumenting a Package X XUse the @code{tst-instrument} command for instrumenting Lisp code. XInstrumenting copies the contents of the current buffer to a new buffer Xnamed for the current buffer concatenated with @samp{-instrumented}. The Xnew buffer is set to emacs-lisp-mode. Instrumenting initializes the Xannotation data base, adds the instrumentation to the copied code, and Xfinally evaluates the entire copied buffer. X X@section Instrumentation Probes X XInstrumentation acts on certain lists that represent functions within a X@code{defun} by adding an instrumentation probe. Candidate lists are Xidentified and Lisp code is added around the list. A list representing a Xfunction that has the form, @samp{(function arg1 arg2)}, after insertion of Xthe instrumentation probe will have the form, @samp{(tst-cover #id X(function arg1 arg2))}. Instrumentation assigns the line number of the Xfunction within the buffer as the #id. X X@section Cover Function X XThe instrumentation probe consists of the invocation of the function X@code{tst-cover} with two arguments, a unique identifier and the function Xthat was instrumented. @code{Tst-cover} uses the identifier as a key into Xthe annotation data base, stores the result of the instrumented function, Xincrements an invocation counter, and returns the result of the Xinstrumented function as its function value. X X X@section Instrumentation Example X XOriginal Code: X X@example X(defun factorial (n) X (let X (result) X (if X (< n 2) X (setq result 1) X (setq result X (* n X (factorial X (1- n))))) X result)) X@end example X XInstrumented Code: X X@example X(defun factorial (n) X (tst-cover 2 (let X (result) X (tst-cover 4 (if X (tst-cover 5 (< n 2)) X (tst-cover 6 (setq result 1)) X (tst-cover 7 (setq result X (tst-cover 8 (* n X (tst-cover 9 (factorial X (tst-cover 10 (1- n)))))))))) X result))) X@end example X X@node analyzing, displaying, instrumenting, top X@chapter Analyzing the Test Results X XThis section explains how to use the @code{analyze} function X X@section Analysis Functions X XThe Coverage Analysis package is designed to run after a series of tests Xhas been performed using the instrumented version of the code under test. XIt retrieves data from the annotation database and detects the following Xtesting anomalies: X X@itemize @bullet X@item XFailure to execute an expression. X@item XFailure to produce more than one value for an expression. X@end itemize XWhen these conditions are detected additional attributes are stored in the Xannotation database in a format suitable for extraction by the display Xcomponent of Test. X XAnalysis is invoked by the command: X X@example X tst-analyze X@end example X XIf an expression is not executed, the attribute ``zero'' is inserted with a Xconstant value. This constant value is defined by the following variable: X@example X tst-anl-zero-counts X@end example X XIf an expression returns a constant value, the attribute ``constant'' is Xinserted with the value of the constant. If an expression is only executed Xonce, it is deemed to return a constant result. X XYou can re-run the analysis function without re-instrumenting the code Xunder test. If, for example, an initial set of tests indicates a number of Xunexecuted expressions, you can run additional tests and analyze the Xcombined results without re-running the complete set of tests. Note Xhowever, that if you re-instrument the code, the annotation database is Xreinitialized and the results of previous tests, including any analysis Xresults will be lost (unless you explicitly save the annotation database Xusing tst-ann-get-db). X X@section Sample Analysis Results X XNormally you would view the results of an analysis using the display Xpackage on-line. However, the following indicates the information produced Xby the analysis. X X@example X# Test analysis of box.el X# (lines which were never evaluated during tests, indicated by X# the string ``NEVER->>'', or which returned the same value X# every time they were evaluated, including the value.) X Xbox.el:31== nil Xbox.el:33== nil Xbox.el:34== nil Xbox.el:35== nil Xbox.el:37== 1 Xbox.el:38== 0 Xbox.el:39== nil Xbox.el:41== nil Xbox.el:42== nil Xbox.el:49== 1 Xbox.el:50== nil X X... X Xbox.el:94== nil Xbox.el:95== nil Xbox.el:109== #<marker at 1 in box.junk> Xbox.el:115== nil Xbox.el:116== t Xbox.el:121== NEVER->> Xbox.el:122== NEVER->> Xbox.el:128== nil Xbox.el:129== nil Xbox.el:134== NEVER->> Xbox.el:135== NEVER->> Xbox.el:140== nil X X... X X@end example X X@node displaying, running, analyzing, top X@chapter Displaying Test Results X XThe @dfn{coverage} component of the Test package is useful for measuring Xhow thoroughly a test script exercises an emacs-lisp package. The Xinteractive Test display mode lets you browse the annotated code. The Xbatch display function creates a summary report which may be browsed later Xusing the Emacs ``compilation'' mode. X X@section Display Mode---creating annotation windows X XEvaluating an instrumented buffer of emacs-lisp code creates a database of X@dfn{annotations} for each instrumented line. The two data stored during Xevaluation are: @samp{count}, the number of times each line has been Xevaluated, and @samp{values}, a list of the resultant values. Analyzing Xthe database adds two new annotations to various lines in the database: X@samp{zero}, the line was never evaluated, and @samp{constant}, the value Xreturned was the same every time it was evaluated. Note that a line Xevaluated exactly once will be flagged as having a constant value. X XDisplay mode puts up these @dfn{annotations} beside the lisp code in one or Xmore @dfn{annotation windows} which are linked to the @dfn{lisp window} so Xthat all windows scroll together. The functions available for creating Xannotation windows are as follows: X X@table @samp X@item C-c c XOpen an annotation window which highlights code which returned the Xsame value during testing (@samp{tst-display-constant}). X X@item C-c z XOpen an annotation window which highlights code which was never Xevaluated during testing (@samp{tst-display-zero}). X X@item M-x tst-display-open-buffer XCreate a buffer containing the database values for a particular attribute Xover all lines of the lisp buffer. Valid attributes are @samp{count}, the Xcount of how many times that line was evaluated, and @samp{values}, a list Xwith each element the result of one evaluation of that line. X X@item M-x tst-display-open-window XOpen an annotation window onto a buffer created by a prior evaluation Xof @samp{tst-display-open-buffer}. X X@item C-c C-h XOpen a help window showing the key bindings for display mode X(@samp{tst-display-mode-help}). X X@item C-c q XExit display mode (@samp{tst-display-mode-exit}). X@end table X XTypically the first thing you will do after entering display mode is to Xtype @samp{C-c c C-c z} to create two annotation windows which point out Xany lines in the lisp buffer which either were never evaluated or always Xreturned the same value. In the first case you probably need to add to Xyour test script to either call an unreferenced function or exercise the Xother side of a conditional expression. The second case may require some Xinvestigation. One possibility is that the line is something like: X@example X(setq very-important-variable nil). X@end example XThis expression, of course, always returns the same value. Another Xpossibility is that the line was only evaluated once, in which case you Xmight want to add to your test script to hit it again. The third Xpossibility is that you overlooked a parameter when composing your test Xscript. X XIf you want to look at the other annotations in the database, you may do so Xwith the commands @samp{M-x tst-display-open-buffer} and @samp{M-x Xtst-display-open-window}. You must use them in that order, and each one Xwill prompt for an ``attribute name'' which may be either of @samp{count} Xor @samp{values}. X X@section Display Mode---moving within annotation windows X XThe annotation windows are kept in step with the lisp window through Xseveral functions which take the place of the usual cursor movement Xcommands while in display mode. These are described below: X X@table @samp X@item C-n XMove down one line vertically in the lisp window and all associated Xannotation windows. On reaching the bottom of the window, scroll windows Xtogether (@samp{tst-display-next-line}). X X@item C-p XMove up one line vertically in the lisp window and all associated Xannotation windows. On reaching the top of the window, scroll windows Xtogether (@samp{tst-display-previous-line}). X X@item C-v XScroll forward in the lisp window and any associated annotation windows Xkeeping them aligned (@samp{tst-display-scroll-up}). X X@item M-v XScroll backward in the lisp window and any associated annotation windows Xkeeping them aligned (@samp{tst-display-scroll-down}). X X@item C-c l XClear screen and redisplay, scrolling the lisp window and any associated Xannotation windows together to center the line containing point X(@samp{tst-display-redraw}). X X@end table X XThe usual cursor up and down keys are @samp{C-n} and @samp{C-p}, Xrespectively. In display mode these move point in all annotation windows Xsimultaneously so that when the limits of the screen are reached all Xwindows will scroll together. Similarly the scroll-up and scroll-down keys X@samp{C-v} and @samp{M-v} are rebound in display mode to functions which Xkeep track of what annotation windows are open and scroll them in step with Xthe lisp buffer. It is still possible for the windows to get out of step X(e.g. after using @samp{M-<}), so the @samp{C-c l} key will resynchronize Xall annotation windows while recentering the line containing point just as Xthe @samp{C-l} key does for a single window. X X@section Batch Display of Analysis X XQuite separate from the display mode is a display function that is designed Xto give a concise summary of test coverage without your interaction. This Xfunction can be run in batch or interactively. (It was originally intended Xfor only batch operation, hence the name.) X X@table @samp X@item M-x tst-display-batch XGenerate a @dfn{compilation style} buffer containing @dfn{zero} and X@dfn{constant} analyses from the database. X@end table X XYou need not be in display mode to use @samp{tst-display-batch}. You are Xmost likely to use it in a test script run from batch mode to dump out a Xsummary of test coverage at the end of a run. However it is also useful Xfrom interactive mode because it takes advantage of the Emacs function X@samp{C-x`} (@samp{next-error}), often used to view compiler error Xmessages. The first invocation of @samp{C-x`} parses the error messages in Xthe buffer named @samp{*compilation*} then places point in one window on Xthe line of code referenced by the error message shown at the top of the Xother window. Successive @samp{C-x`} keystrokes advance to successive Xerror messages and the corresponding lines in the code buffer. X XThe @samp{M-x tst-display-batch} command writes into the X@samp{*compilation*} buffer a line for every line in the lisp buffer that Xhas either a @samp{zero} or @samp{constant} annotation. Subsequent uses of X@samp{C-x`} advance to the next error and the corresponding line in the Xlisp buffer. X X@section The Example X XThe following is the output of the evaluation of @samp{tst-display-batch} Xin our example test script. Notice that some deficiencies in the test Xscript are pointed out: error conditions were not exercised (see lines X121-122 and 134-135 in Appendix A); and not enough different input texts Xwere ``boxed'' to really stress the package under test (e.g. line 40, input Xtext always same width). X X@example X# Test analysis of box.el X# (lines which were never evaluated during tests or returned X# the same value every time they were evaluated.) Xbox.el:31== nil Xbox.el:33== nil Xbox.el:34== nil Xbox.el:35== nil Xbox.el:37== 1 Xbox.el:38== 0 Xbox.el:39== nil Xbox.el:40== 41 Xbox.el:41== nil Xbox.el:42== nil Xbox.el:43== 42 Xbox.el:47== 44 Xbox.el:49== 1 Xbox.el:50== nil Xbox.el:51== nil Xbox.el:52== nil Xbox.el:53== nil Xbox.el:55== nil Xbox.el:56== nil Xbox.el:57== nil Xbox.el:58== nil Xbox.el:59== nil Xbox.el:62== nil Xbox.el:63== nil Xbox.el:64== nil Xbox.el:65== nil Xbox.el:67== nil Xbox.el:68== nil Xbox.el:69== 1 Xbox.el:70== 12 Xbox.el:71== nil Xbox.el:72== nil Xbox.el:73== nil Xbox.el:74== 12 Xbox.el:75== nil Xbox.el:80== nil Xbox.el:91== nil Xbox.el:93== nil Xbox.el:94== nil Xbox.el:95== nil Xbox.el:109== #<marker at 6 in box.junk> Xbox.el:115== nil Xbox.el:116== t Xbox.el:121== NEVER->> Xbox.el:122== NEVER->> Xbox.el:128== nil Xbox.el:129== nil Xbox.el:134== NEVER->> Xbox.el:135== NEVER->> Xbox.el:140== nil Xbox.el:141== nil Xbox.el:143== nil Xbox.el:144== t Xbox.el:145== NEVER->> Xbox.el:147== t Xbox.el:149== nil Xbox.el:150== nil Xbox.el:151== nil Xbox.el:153== nil Xbox.el:154== nil Xbox.el:157== nil Xbox.el:159== nil Xbox.el:160== #<marker at 6 in box.junk> X X@end example X X@node running, epilogue, displaying, top X@chapter Running Test in Batch Mode X X@section Preparation X XBefore running a set of tests, you must prepare the following items: X@itemize @bullet X@item XA package to test. X@item XA test script, usually written as a single function. X@item XA set of precondition states---one for each test (although many Xtests may share the same precondition). X@item XA set of expected postcondition states---one for each test (some Xsharing is possible). X@end itemize XTest scripts are discussed in the next section. X XYou will probably want to use the state-capturing functions (e.g., X``tst-capture-state'', ``tst-write-state-to-file'') to prepare and save Xstates in files. In doing this, strive for minimality of states. That is, Xavoid complicating the states with extra components, particularly buffers Xand windows, that do not contribute to the functionality being tested. XThis will make it easier to read and comprehend the testing states, and Xwill result in more efficient test scripts. (The instrumentation and Xanalysis components of Test are designed to assist you in discovering the Xinadequacies of your test scripts. It is best to start with too little Xtesting, and add additional tests as the analysis dictates.) X XBelow is an example scenario for preparing the states used in a testing Xscript discussed later. Note that the functions would be invoked Xinteractively, even though they are shown in ``program invocation'' style. X@example X(load "test.el") X(load "box.el") X(find-file "sample") X... go to an appropriate location ... X(set-mark) X... go to an appropriate location ... X(tst-capture-state-to-file "pre-box") X(box-region) X(tst-capture-state-to-file "exp-box-1") X(unbox) X(tst-capture-state-to-file "exp-unbox-1") X(box-region t) X(tst-capture-state-to-file "exp-box-2") X(unbox t) X(tst-capture-state-to-file "exp-unbox-2") X@end example XNote that in this scenario the expected result of applying a function is Xcreated by applying the function to the precondition state. It is hard to Ximagine how the test could fail, but it might for unexpected reasons. A Xbetter method of creating the test states is to use an independent method, Xsuch as repeated application of simpler functions (e.g., ``insert'', X``next-line''). Still, ``safe'' tests, such as that shown above, are Xuseful for regression testing. X X@section Batch Test Scripts X XA test run usually consists of the following steps: X@enumerate X@item XRead testing states from files. X@item XInitialize the testing environment: X@enumerate X@item XFind the object package (the code to be tested). X@item XInstrument the package. X@end enumerate X@item XRun each test: X@enumerate X@item XAchieve the precondition state of the test. X@item XExecute the function to be tested. X@item XCapture the postcondition state. X@item XCompare the postcondition state to the expected postcondition state. X@end enumerate X@item XSave the results of testing in a file. X@item XAnalyze the testing (filter the collected data) and save the results. X@end enumerate XNote that a set of tests may reuse testing states, especially the Xpreconditions of tests. That is why it is wise to read all of the needed Xtesting states from files first. Also, it is good style to minimize state Xchanges during testing (except for the execution of tested functions, of Xcourse), so that the test script may be modified without unforeseen side Xeffects. X XBelow is an example test script for testing functions ``box-region'' and X``unbox'' in package ``box.el''. This script does not contain enough Xtests, which the instrumentation should expose. X X@example X(load "test.el") X X(defun script () X (let (post-state pre-box exp-box-1 exp-box-2 X pre-unbox exp-unbox-1 exp-unbox-2) X ; Load states from files X (tst-read-state-from-file 'pre-box "pre-box") X (tst-read-state-from-file 'exp-box-1 "exp-box-1") X (tst-read-state-from-file 'exp-box-2 "exp-box-2") X (tst-read-state-from-file 'pre-unbox "pre-unbox") X (tst-read-state-from-file 'exp-unbox-1 "exp-unbox-1") X (tst-read-state-from-file 'exp-unbox-2 "exp-unbox-2") X ; Instrument package X (find-file "box.el") X (tst-instrument) X ; Initialize test environment X (find-file "sample") X ; Tests X ; TEST box-1 X (tst-achieve-state pre-box) X (box-region nil) X (tst-capture-state 'post-state) X (tst-equ-state post-state exp-box-1) X ; TEST box-2 X (tst-achieve-state pre-box) X (box-region t) X (tst-capture-state 'post-state) X (tst-equ-state post-state exp-box-2) X ; TEST unbox-1 X (tst-achieve-state pre-unbox) X (unbox nil) X (tst-capture-state 'post-state) X (tst-equ-state post-state exp-unbox-1) X ; TEST unbox-2 X (tst-achieve-state pre-unbox) X (unbox t) X (tst-capture-state 'post-state) X (tst-equ-state post-state exp-unbox-2) X ; Results of tests X (set-buffer "*equal-log*") X (save-file "equal-log-sample") X ; Analysis of testing X (tst-analyze) X (tst-display-batch) X ) ; let X) ; defun script X@end example X X@section Batch Invocation of Test X XTo invoke the test script from batch, use the following Unix command: X@example X% gnuemacs -batch -l script.el -f script -kill & X@end example XNote that the test script explicitly loads ``test.el'' at the beginning Xof ``script.el''. X X XAn instrumented package runs considerably slower than an uninstrumented Xpackage. It is a good idea to test your script without instrumentation X(and probably without many of the tests). Once you are satisfied with the Xscript, run it at lower priority and/or at non-peak hours to avoid Xinconveniencing other users. X X@section Advanced Batch Use -- Filtering X XThe state information available for ``capture'ing and ``achieve''ing is Xusually more than needed for any particular test. You can improve the Xperformance of testing by filtering out the unneeded components of states. XThere are mechanisms for doing this before and after testing: X@itemize @bullet X@item XThe ``capture'' functions use global options to determine the Xcomponents to capture. X@item XThe ``achieve'' functions are similarly parameterized. X@item XThe ``equality'' functions use global options to determine which components Xto compare. Also, they can be weakened (i.e. return ``true'' more often) Xthrough the use of hooks and replacement functions. X@item XThe ``analysis'' functions reduce the instrumentation data to two special Xcases: lack of evaluation and ``constant value across testing''. X@item XThe ``*equal-log*'' buffer may be viewed in ``outline'' mode, with Xuninteresting cases suppressed through elision. X@item XThe batch display function produces output that may be viewed with the X``next-error'' function of ``compilation''. X@end itemize XIn using these filters you should be aware that too much filtering defeats Xthe purpose of testing. X X@node epilogue, example, running, top X@chapter The Epilogue X X@section How to Extend the Package X X@subsection Equality Tests X XIf you want to extend the equality package to compare other aspects of your Xenvironment, simply write your own comparison function and add it to the X'function-vector' for the appropriate area. For example, if you were Xinterested in comparing the @code{foobar} attribute of buffers, first write Xa function, @code{tst-equ-foobar}, and then add it to the list of functions Xexecuted for a buffer, @code{tst-equ-buff-state-functions}. X XSimilarly, if you wanted to test the @code{foo} attribute of a state, add Xthe new function to the list of functions executed for a state, X@code{tst-equ-state-functions}. X XYou can also extend the equality tests by the use of hooks. The method is Xdefined in more detail in the section on Inequality. X X@subsection Instrumentation X X@subsection Display Functions X XThe interactive display mode attempts to provide the fiction of annotation Xwindows which are part of the code window. In retrospect it may be that Xsimply inserting the text into a new buffer along with the code would be as Xuseful. It also may be that the batch style report in conjunction with the X``compilation'' error parsing is just as useful in practice. X XThere are certainly other types of annotation which could usefully be Xdisplayed, like the number of times each line was executed, etc. X XIt might make sense to bind @samp{tst-display-batch} to a key in X@code{test-mode}. It also might make sense to remove the binding for X@code{tst-analyze} and call it only from within the display functions. X X@subsection Annotation Functions X XThe annotation database is accessed twice every time an instrumented line Xof code is evaluated. This code contributed substantially to the slow-down Xof instrumented code and time spent making it more efficient would be Xrewarding. X X@subsection Analysis Functions X XIf you want to add further functions to the Analysis Package, you will have Xto modify the main function, ``tst-analyze'', which invokes the individual Xanalysis functions. It currently invokes ``tst-anl-zero-counts'' and X``tst-anl-constant-values''. Your new function should use the same idiom Xas these functions, i.e., use ``tst-ann-get-lines'' to retrieve the list of Xlines for which annotation holds information and then use ``mapcar'' to Xapply your single line analysis function to each line's data. You can then Xuse ``tst-ann-put'' to store your results. X X@node example, , epilogue, top X@appendix The Complete Example X@include example SHAR_EOF if test 60887 -ne "`wc -c < 'test.texinfo'`" then echo shar: "error transmitting 'test.texinfo'" '(should have been 60887 characters)' fi fi exit 0 # End of shell archive -- Rich $alz Cronus Project, BBN Labs rsalz@bbn.com Moderator, comp.sources.unix sources@uunet.uu.net